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Introduction 


The  project  aims  to  develop  a  battery  of  tests  to  diagnose  visual  dysfunc¬ 
tions  in  cognitively  impaired  observers  based  on  their  oculomotor  reflexes. 
The  oculomotor  reflexes  provide  a  simple  and  robust  method  to  study  vision 
in  passive/immature/impaired  observers.  For  example,  oculomotor  reflexes 
are  widely  used  to  study  vision  in  infants  and  in  marketing  research.  The  pro¬ 
posed  method  is  versatile,  applies  to  a  wide  range  of  visual  mechanisms,  and 
can  provide  both  qualitative  (yes/no)  and  quantitative  (degree)  estimates  of 
the  visual  loss.  The  project  embodies  a  full  research  and  development  cy¬ 
cle,  from  establishing  the  most  effective  stimuli  for  various  kinds  of  visual 
dysfunctions  to  the  design  of  a  computerized  testing  kit  suitable  for  use  by 
non-specialists.  After  a  short  training  course,  the  kit  could  be  used  at  army 
bases  and  local  hospitals  to  make  a  quick  initial  diagnosis. 
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Figure  1:  Gantt  chart  for  the  project. 

As  specified  in  the  Statement  of  Work,  the  main  goal  of  the  second  12- 
months  period  of  the  project  was  to  make  the  tools  developed  in  the  first  aim 
available  to  non-specialists.  To  this  end,  a  computerized  testing  kit  based 
on  an  autmatic  eye-event  registration  algorithm  was  developed.  The  kit  is 
used  to  present  visual  stimuli,  collect  eye-position  and  pupil-size  data  and 
search  the  data  for  relevant  ocular  reflexes  evoked  in  a  real-time  automated 
regime.  Based  on  the  analysis,  an  adaptive  algorithm  chooses/adjusts  the 
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test  stimuli  for  the  next  round  of  stimulation  until  a  certain  reliability  level  is 
achieved  and  the  battery  of  tests  is  exhausted.  At  the  moment  the  test  results 
are  presented  as  percent  correct  scores.  Once  the  full  statistics  on  normal 
observers  and  observers  with  visual  dysfunctions  is  collected  the  results  will 
be  presented  in  the  form  of  probability  scores  for  different  kinds  of  visual 
dysfunctions. 

Figure  1  shows  a  Gantt  chart  indicating  project  tasks  and  their  current 
state  of  completion.  Overall,  all  the  current  goals  were  met:  the  hardware  kit 
was  augmented  for  real-time  data  exchange  between  the  video-presentation 
and  data-collection  laptops,  a  software  counterpart  for  the  data  exchange 
was  developed  and  tested,  a  large  portion  of  the  pilot  stimuli  were  modi¬ 
fied  to  their  final  form,  algorithms  for  eye-event  registration  were  improved 
and  implemented  in  C++  within  the  same  software  package  as  the  stimulus 
generation  and  data  exchange.  These  tasks  are  described  in  detail  in  the 
following  sections. 


Hardware  development 


Figure  2:  The  complete  hardware  kit.  The  data-collection  laptop  is  on  the  left,  the 
video-presentation  and  data  analysis  laptop  is  on  the  right.  The  eye-tracker  is  in 
the  center,  its  electronics  (MiniLab)  is  in  the  back,  the  head-mounted  eye  cameras 
and  1R  mirror  attached  to  a  safety- google  frame  are  in  the  front.  The  Keyspan 
serial-USB  adapter  connecting  the  eye-tracker  to  the  data-analysis  laptop  can  be 
seen  in  the  center. 

The  hardware  kit  was  completed  in  the  first  quarter.  A  photograph  of 
the  full  kit  (the  head-mounted  version)  is  shown  in  Figure  2.  To  ensure 
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maximum  performance  two  computers  (laptops)  are  used.  One  computer  is 
dedicated  to  controlling  the  eye-tracker.  Stimulus  generation,  presentation, 
data  collection,  and  data  analysis  are  carried  out  by  another  computer.  The 
two  computers  communicate  via  serial  connection.  Eye  images  are  sampled  at 
120  Hz  by  the  ISCAN  Inc.  eye-tracker.  120  Hz  was  proven  to  be  an  adequate 
sampling  frequency  for  our  purposes.  Video  components  of  the  eye-tracker 
include  miniature  eye-  and  scene-cameras,  infrared  (IR)  illuminator  LED,  and 
an  IR  mirror  all  mounted  on  a  safety-goggles  frame  positioned  on  a  subject’s 
head.  The  electronic  components  (MiniLab)  include  hardware-implemented 
algorithms  for  segmenting  the  pupil  image  from  the  eye  image  and  calculating 
the  pupil’s  position  and  size.  MiniLab  also  includes  hardware  for  the  serial 
data  output.  The  eye-tracker  is  controlled  from  the  laptop  shown  on  the 
left.  ISCAN  software  installed  on  the  laptop  provides  an  intuitive  graphical 
user  interface  (GUI)  allowing  to  control  the  eye-tracker  in  a  simple  fashion. 
In  particular,  it  is  used  to  optimize  the  camera  settings  for  various  lighting 
conditions  and,  if  required,  for  glasses  or  contact-lenses  worn  by  the  subject. 
The  laptop  shown  on  the  right  is  used  for  visual  stimulation  and  online 
analysis  of  the  eye-tracking  data.  Visual  stimuli  are  presented  either  on  the 
laptop  screen  or  on  a  dedicated  monitor  (not  shown).  The  eye-tracking  data 
is  received  in  real-time  regime  from  the  MiniLab  electronics  via  a  6  ft.  serial 
cable.  The  data  is  received  on  a  LTSB  port  of  the  laptop  via  a  Keyspan 
serial-LTSB  adapter  (model  USA-19HS). 

The  head-mounted  version  of  the  eye-tracker  can  be  mounted  on  a  base¬ 
ball  cap  if  the  safety-goggles  frame  is  not  tolerated  by  the  subject.  If  neither 
version  of  the  head-mounted  device  can  be  safely  worn  by  the  subject  the 
remote  (desktop  mounted)  eye-tracker  with  automatic  head  tracking  can  be 
used.  The  remote  eye-tracker  will  also  be  used  for  detecting  dysfunctions  in 
binocular  convergence  for  all  subjects,  because  it  allows  to  track  both  eyes 
simultaneously. 

The  hardware  kit  was  tested  in  both  the  Pi’s  laboratory  and  in  field 
conditions.  In  the  latter  case  the  kit  was  taken  to  the  Brown  LIniversity, 
where  a  subject  with  recent  TBI-related  visual  dysfunctions  was  tested.  The 
kit  weighed  about  20  pounds  and  was  carried  in  a  medium-sized  backpack. 
It  took  less  then  10  minutes  to  assemble  the  kit  and  to  get  it  ready  for  the 
test. 


Visual  tests  development 

In  the  third  quarter  the  pilot  stimuli  developed  and  tested  during  the  first 
year  were  improved  to  produce  more  robust  eye  events  suitable  for  automated 
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Reflex 

Stimulus 

Functions 

Visual  Nystagmus 

luminance  grating 

equiluminant  grating 

contrast  gratings  at  3  spatial  scales 

semi-coherently  drifting  dots 

low-level  motion 

color  vision 

visual  acuity 

high-level  motion 

Orienting  Reflex 

contour  /  texture  boundary 

odd  image  (animal  among  males) 

seminude  female  vs.  ’boring’  male 

low-level  object  processing 

mid-level  object  processing 

high-level  object  processing 

Pupillary  reflex 

mild  curse  in  a  random  word  stream 

odd  word  within  a  sentence 

mid-level  reading 

high-level  reading 

Table  1:  Oculomotor  reflexes,  the  stimuli  used  to  evoke  the  reflexes,  and  the 
corresponding  tested  visual  functions. 


detection.  The  ocular  reflexes,  the  stimuli  used  to  evoke  these  reflexes,  and 
the  corresponding  tested  visual  functions  are  presented  in  Table  1. 

We  developed  a  new  stimulus  for  visual  nystagmus  to  test  visual  mo¬ 
tion  processing  in  the  presence  of  incoherent  motion  noise.  The  drifting 
equiluminant  color  grating  stimulus  used  for  testing  color  deficiencies  was 
redesigned  to  make  the  evoked  visual  nystagmus  much  more  robust.  Because 
the  absolute  gaze  is  not  measured  in  our  paradigm  (this  would  require  a  gaze 
calibration,  involving  significant  cooperation  of  the  subject)  the  orienting  re¬ 
flex  has  to  be  measured  with  respect  to  an  undefined  gaze  position  at  the 
time  of  the  stimulus  onset.  To  attract  the  gaze  to  the  center  of  the  screen 
a  disk  contrast-reversing  for  500  msec  is  now  presented  at  the  center  of  the 
screen  1000  msec  before  the  stimulus  onset.  In  addition,  the  odd-ball  im¬ 
age  and  contour/boundary  stimuli  used  to  evoke  the  orienting  reflex  were 
simplified  to  make  the  evoked  eye  movements  less  dependent  on  the  gaze 
position  at  the  time  of  the  stimulus  onset.  To  this  end,  the  odd  image  or  the 
contour/boundary  was  positioned  in  one  of  the  4  quadrants  on  the  screen 
(compared  to  8  such  positions  used  in  pilot  stimuli).  The  test  was  considered 
passed  if  the  subject’s  gaze  stayed  in  the  ’interesting’  quadrant  significantly 
longer  than  in  the  remaining  quadrants.  The  odd-word  stimulus  originally 
designed  to  test  the  ability  to  read  and  comprehend  isolated  words  was  ex¬ 
tended  to  include  complete  sentences,  where  the  ability  to  understand  the 
logical  structure  of  a  written  text  is  also  tested. 
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Software  development 

The  four  eye-event  filters  (blinks,  saccades,  nystagmus,  and  pupillary  reflex) 
prototyped  in  MATLAB  in  the  course  of  the  previous  year  were  implemented 
in  C++.  This  allowed  us  to  increase  the  filters  speed  and  to  unite  stimulus 
generation  and  data  analysis  into  a  feedback  loop  within  the  same  application 
running  on  the  data-analysis  laptop.  The  filters  are  based  on  measuring  and 
thresholding  eye-movement  velocities  and  pupil  area  changes.  From  the  start 
the  filters  were  designed  to  be  used  with  real-time  data:  they  are  fast  and  do 
not  require  large  amounts  of  data.  Less  than  a  second  of  data  is  sufficient  to 
detect  saccades  and  pupillary  reflex,  and  several  seconds  of  data  -  to  detect 
visual  nystagmus  which,  typically,  lasts  for  3  -  5  seconds  at  a  time.  In  the 
third  quarter  we  thoroughly  redesigned  the  nystagmus  and  pupil  dilation 
filters  to  make  their  output  more  robust  to  subjective  variations.  To  this 
end  we  developed  a  new  algorithm,  where  the  eye-tracking  data  is  analyzed 
at  different  time  scales  and  then  the  resulting  filter  outputs  are  integrated 
across  the  scales.  We  also  completely  redesigned  the  filter  used  to  measure 
the  orienting  reflex  (eye-movement  and  the  following  viewing  of  an  interesting 
object).  Instead  of  relying  on  the  direction  of  the  first  saccade  we  used  a 
different  approach  based  on  measuring  how  long  the  subject’s  gaze  stayed 
within  the  area  of  interest  compared  to  the  rest  of  the  scene.  This  measure 
proved  to  be  more  robust  to  the  effect  of  the  (somewhat  unpredictable)  gaze 
position  at  the  beginning  of  the  trial. 

Figure  3  illustrates  the  new  filters  performance.  Raw  eye-tracking  data  (a 
single  trial  in  each  case)  are  shown  by  the  black  symbols,  the  outputs  of  the 
corresponding  filters  are  marked  in  red.  Panel  a  illustrates  a  filter  response 
to  a  significantly  longer  viewing  of  an  odd  (animal  face)  image  shown  in  the 
bottom-right  quadrant.  Panel  b  illustrates  a  similar  paradigm,  where  a  semi¬ 
nude  female  image  shown  in  the  left  half  of  the  screen  was  viewed  longer 
than  a  ’boring’  male  image  in  the  right  half.  The  filter  response,  again, 
indicates  a  significantly  longer  viewing  of  the  ’interesting’  image.  Panel  c 
illustrates  a  robust  visual  nystagmus  arising  in  response  to  the  stimulus  where 
a  coherent  drift  of  some  dots  is  masked  by  incoherent  random  movements  of 
the  remaining  dots.  Panel  d  illustrates  a  pupillary  dilation  in  response  to  a 
mild  curse  word  suddenly  appearing  within  a  sequence  of  ’ordinary’  words 
presented  on  the  screen. 

Currently  we  are  working  on  tweaking  the  visual  stimuli  and  the  respec¬ 
tive  filters  to  make  the  filter  performance  maximally  independent  of  subjec¬ 
tive  variations  in  the  eye-tracking  data.  This  is  most  crucial  while  detecting 
the  pupillary  response  to  a  surprising  stimulus  (e.g.,  a  curse  word),  where 
the  latency,  amplitude,  and  duration  of  the  pupil  dilation  were  found  to  vary 
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significantly  from  trial  to  trial  and  also  among  subjects. 


Testing 

Once  the  Northeastern  University  IRB  and  MEDCOM  USAMRMC  approvals 
for  work  with  human  subjects  were  renewed  in  October  2011  we  collected  and 
analyzed  data  for  twenty  five  normal  observers  and  two  observers  with  pos¬ 
sible  visual  dysfunctions.  Fifty  more  observers  will  be  tested  in  the  fourth 
quarter.  The  large  number  of  observers  is  necessary  to  accumulate  enough 
statistics  to  estimate  the  distribution  of  test  scores  (percent  correct)  among 
normal  observers.  This  distribution  will  be  used  to  convert  the  test  scores  to 
the  probability  of  visual  dysfunctions. 

Northeastern  University  undergraduate  students  of  both  sexes  18  -  25 
years  of  age  with  normal  or  corrected  vision  were  recruited  as  normal  sub¬ 
jects.  The  subjects  were  inexperienced  observers  naive  to  the  purpose  of  the 
test.  The  only  instructions  received  from  the  experimenter  were  to  attend  to 
the  screen.  The  subjects  and  experimental  conditions  are  meant  to  simulate 
testing  cognitively  impaired  soldiers.  We  also  carried  out  pilot  tests  of  two 
subjects  with  recent  history  of  visual  dysfunctions.  Subject  SI  (a  female,  37 
years  old)  had  a  severe  traumatic  brain  injury  8  months  prior  to  the  test.  As 
a  result,  her  control  of  eye  movements  was  initially  disrupted  to  the  extent 
that  for  several  months  she  could  not  read  at  all,  could  not  do  her  work  or 
travel.  At  the  time  of  the  test  she  described  her  visual  functions  as  almost 
normal.  Superficially,  her  eye  movements  also  appeared  normal.  Subject  S2 
(a  male,  70  years  old)  had  a  foveal  retinal  detachment  in  both  eyes.  A  year 
prior  to  the  test  he  was  operated.  The  first  operation  was  unsuccessful  in  the 
right  eye,  but  after  the  second  operation  he  reported  his  vision  to  be  mostly 
restored  in  both  eyes. 

Experimental  sessions  lasted  6  minutes  during  which  80  visual  stimuli 
were  shown.  Subjects  were  viewing  stimuli  on  a  21”  LCD  monitor  from  a 
distance  of  60  -  80  cm.  The  head-mounted  version  of  the  eye-tracker  was 
used  for  all  subjects,  pupil  (left  eye)  position  on  the  camera’s  CCD  was 
sampled  at  120  Liz.  The  data  were  collected  and  analyzed  by  the  developed 
software  filters  in  real  time  and  then  saved  on  the  hard  drive  of  the  data- 
analysis  laptop.  A  fragment  of  the  data  and  the  accompanying  automatic 
eye-event  segmentation  for  one  subject  are  shown  in  Figure  3.  Compared 
to  normal  subjects,  SI  showed  lower  scores  in  the  odd-word  experiment. 
This  suggests  that  Si’s  reading  speed  was  not  restored  to  its  normal  rate. 
Subject  S2  showed  no  equiluminant  (color-defined)  nystagmus  and  very  low 
scores  for  contrast-defined  nystagmus.  This  indicates  that  S2’s  foveal  vision 


was  relatively  poor,  lacking  in  color  sensitivity  and  acuity.  Although  at  the 
current  stage  of  the  project  it  is  too  early  to  draw  definite  conclusions  about 
the  effectiveness  of  our  approach,  these  pilot  tests  make  us  optimistic. 
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Figure  3:  Eye  tracking  data  and  the  corresponding  filter  outputs.  Time  from  the 
stimulus  onset  is  plotted  along  the  x-axis  in  panels  b  -  d.  (a)  An  animal  face 
was  shown  in  the  bottom- right  quadrant,  3  human  male  faces  were  shown  in  the 
remaining  quadrants.  Gaze  point  readings  (x  and  y  coordinates  on  the  camera’s 
CCD)  are  marked  by  the  black  dots.  Only  3  images  were  explored  by  the  subject’s 
gaze.  The  red  circle  indicates  a  significantly  longer  viewing  of  the  bottom-right 
image  detected  by  the  filter,  (b)  a  semi- nude  Marilyn  Monroe  image  was  shown  in 
the  left  half  of  the  screen,  a  male  figure  in  a  suit  was  shown  in  the  right  half.  Gaze 
horizontal  position  is  plotted  along  the  y-axis.  A  significantly  longer  viewing  of 
the  left  image  detected  by  the  filter  is  indicated  by  the  red  arrow,  (c)  For  the  first 
quarter  of  the  trial  randomly  placed  dots  were  moving  in  a  ’random  walk’  fashion, 
afterwards  25%  of  the  dots  were  also  drifting  to  the  right.  Gaze  horizontal  position 
is  plotted  along  the  y-axis.  The  red  bar  indicates  a  visual  nystagmus  event  detected 
by  the  filter,  (d)  A  mild  curse  word  was  displayed  on  a  white  background  at  the 
beginning  of  the  trial.  Pupil  area  is  plotted  along  the  y-axis.  The  red  bar  indicates 
pupil  dilation  detected  by  the  filter.  The  pupil  contraction  at  the  beginning  of  the 
trial  resulted  from  the  response  to  the  white  background. 
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Key  Research  Accomplishments 


•  The  hardware  kit  was  finalized,  its  components  and  settings  were  tested 
and  optimized. 

•  Real-time  data  exchange  between  the  eye-tracker  and  the  data-analysis 
computer  was  implemented  and  tested. 

•  Eye-event  filters  were  redesigned  to  be  less  sensitive  to  individual  vari¬ 
ation  in  ocular  reflexes.  The  filters  were  implemented  in  C++  and 
incorporated  into  the  software  running  on  the  visual  stimulation  and 
data  analysis  laptop.  The  real-time  processing  of  the  data  will  be  used 
to  make  the  testing  adaptive. 

•  Visual  stimuli  were  improved  and  simplified  to  produce  more  robust 
visual  responses. 

•  The  complete  kit  was  tested  on  25  normal  observers  naive  to  the  pur¬ 
pose  of  the  test  and  two  observers  with  possible  visual  dysfunctions 
resulting  from  recent  TBI  and  retinal  detachment  injuries. 

•  When  compared  with  normal  subjects  the  two  observers  with  possible 
visual  dysfunctions  showed  significantly  lower  scores  which  appear  to 
be  in  agreement  with  their  prior  symptoms  and  clinical  histories. 
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Reportable  Outcomes 


There  were  no  reportable  outcomes  yet. 
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Conclusions 


We  significantly  improved  the  battery  of  tests  designed  to  diagnose  various 
visual  dysfunctions.  All  hardware  and  software  components  of  data  acqui¬ 
sition  and  analysis  pipeline  are  in  place  now  allowing  data  collection  and 
analysis  in  real-time  regime.  The  visual  stimulation  and  data  collection  units 
were  tested  on  25  naive  observers  with  normal  vision  and  2  observers  with 
possible  visual  dysfunctions.  We  used  the  collected  data  to  make  a  pilot 
comparison  of  oculomotor  responses  between  normal  subjects  and  subjects 
with  visual  dysfunctions.  The  results  indicated  that  the  adopted  paradigm 
could  successfully  diagnose  visual  dysfunctions.  This  mostly  completes  the 
second  phase  of  the  project  and  allows  us  to  start  the  last  phase,  which 
mainly  focuses  on  the  two  goals:  testing  a  large  number  of  normal  subjects 
to  establish  the  distribution  of  responses  for  people  with  normal  (reference) 
visual  functions,  and  testing  subjects  with  visual/cognitive  dysfunctions  to 
validate  the  proposed  approach. 
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